Repository navigation
Log each Sidekiq line once, with the job's ID, and write logged Hashes at the top level - #17
Merged
Merged
Conversation
In a Sidekiq server process, Sidekiq's Rails integration calls Rails.logger.broadcast_to(Sidekiq.logger) after initialization. LogStruct's logger re-sent every String message to its broadcasts, and both loggers write to the same SemanticLogger appenders, so each Rails.logger call inside a job was written twice: once as an app line and once as a Sidekiq line. broadcast_to now ignores SemanticLogger loggers. Fixes #13
The Sidekiq logger put the live Sidekiq::Context hash into each log struct. SemanticLogger formats on its appender thread, so for a fast job it could still be iterating that hash when Sidekiq's JobLogger added :elapsed after the job finished. The resulting "can't add a new key into hash during iteration" error was raised after the job had succeeded, so Sidekiq marked it as failed and retried it. Fixes #14
Sidekiq's own log lines carry the job ID and class in ctx, but lines the job itself writes with Rails.logger don't, so there was no way to pull every line for one job run out of the logs. Until the broadcast fix, the duplicate Sidekiq copy of each line was the only one with a job ID. A Sidekiq server middleware, first in the chain, now pushes the job's jid and class as SemanticLogger named tags for the job's duration, and the formatter writes them to ctx unless the line already has one. ActiveJob jobs show their own class rather than the Sidekiq wrapper, matching Sidekiq's context.
The logger passed a Hash to SemanticLogger as `payload: hash`, which
SemanticLogger stores as {payload: hash}, so Rails.logger.info(src:
"rails", evt: "rate_limit", ...) was written as
{"payload":{"src":"rails",...},"src":"app","evt":"log"}. Fields that
log searches and metric filters expect at the top level were nested
under "payload", and the line's own src and evt were overridden. Hashes
are now passed as the payload itself; structs are unchanged.
A logged Hash may set its own src and evt. Its ts and lvl are reported
as LogStruct errors and skipped, since the line's timestamp and level
come from the logger. During a Sidekiq job, a Hash's own ctx keeps its
keys and gains the job's jid and class.
Sidekiq's default error handler logged three warn lines for each failed job: the job hash as a JSON string inside msg (with the job's arguments, which LogStruct's filters can't reach there), the error, and the backtrace. LogStruct replaces it with one structured line (src sidekiq, evt error) with the error class, message, backtrace, jid, job class, queue and retry count, and no arguments. ActiveJob jobs' errors are left to the ActiveJob integration, which already logs them. Error handlers the app added stay.
|
Warning Review limit reached
This review includes 9 billable files and costs up to $2.25.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 38 minutes for your next included review. View limit details
Comment |
This was referenced Oct 17, 2026
Merged
ndbroadbent
changed the base branch from
nathan/filtering-security-fixes
to
main
October 17, 2026 23:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of twelve stacked PRs for 0.2.0.
What this fixes
Rails.loggertoSidekiq.logger, and both write to the same SemanticLogger appenders.broadcast_tonow ignores SemanticLogger loggers.Sidekiq::Contexthash into each entry, and SemanticLogger formats on its own thread, so Sidekiq adding:elapsedraised "can't add a new key into hash during iteration". It now logs a copy.jidand class inctx, so you can pull every line for one job run out of the logs. A Hash logged with its ownctxis merged with the job's.payload. A Hash can set its ownsrcandevt;tsandlvlare reserved, so they're reported as an error and dropped.jid, class, queue and retry count, instead of Sidekiq's default handler's three text lines with the job's arguments in a JSON string. ActiveJob jobs' failures are left to the ActiveJob integration.Addresses #13 and #14.
Testing
Every regression test fails without its fix. Sorbet and the unit suite (293 tests) pass at this branch's tip.
Stack for 0.2.0, merged in order: #16 → #17 → #18 → #19 → #20 → #21 → #22 → #23 → #24 → #25 → #26 → #27